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(57) Abstract 

An aoparatus and method for transmitting prosthetic information to the brain contains an array of sensoiy elements (12) 
that re^ve energy from an external stimulus and piooess those signak via neural filters (24, 26 and 28) and neural waveforms to 
orodl^a Dulse or "spike" train that is temporally encoded with infoimation that is functionally related to the external stimulus. 
ThVsimulated spike train, when appUed to an appropriate area ofthe brain, produces perceptions that are funrtionally related to 
the sensed external stimuli so that a subject can discriminate between different spike trains representative of different external su- 
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APPARATUS AND METHOD FOR TRANSMITTING PROSTHETIC 
INFORMATION TO THE BRAIN 
BACKGROXJND OP THE INVENTION 
The present invention relates to transmission of 
5 sensory information obtained by a prosthetic device to 
the brain to create a sensory perception. In particular, 
the present invention relates to the transmission of 
visual, audile, or tactile information to the brain in an. 
encoded form that causes a functionally related 
10 perception on the part of the subject. 

In normal circumstances, a healthy hximan being 
receives a stimulus from the external environment that is 
detected by the appropriate type of receptor. For 
example, photoreceptors within the eye detect light with 
15 each photoreceptor converting the received stimulus into 
neuron impulses. This conversion of the stimulus energy 
into neuron impulses takes place in one or more neurons 
that are associated with the receptor. Each neuron then 
emits a neuron spike sequence that is used by the brain 
20 to obtain sensory perception. The combination of many 
neurons, each transmitting many neuron spike trains 
provide a complete or global sensory perception. 

It is known that a simulated neuron spike or pulse 
train can produce a sensory perception in a subject when 
25 applied to selected areas of the brain, spinal column, or 
nerves of a subject. Additionally, the brain has been 
•mapped' so that specific locations on or within the 
brain are associated with specific sensory perceptions. 
Placement of an electrode at an appropriate location and 
30 stimulated with electrical pulses will produce a sensory 
perception. For example, an electrode placed on the 
striate cortex and to which a single simulation neuron 
impulse is transmitted will cause some type of visual 
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perception. 

Repeated pulses to a specific location, such as the 
striate cortex, will produce some type of perceivable 
image. However, meaningful information as to form, 
5 luminescence, and color is not conveyed. It is presently 
believed that the informational content for all 
dimensions of a sense have their basis in a coding scheme 
that is based upon the number of neuron impulses that 
occur during a predetermined interval, for example, a 

10 greater number of neuron impulses are believed to be a 
function of a more intense perceived parameter. However, 
this coding scheme only allows one dimension of a sense 
to be coded by each neuron. Proponents of this coding 
scheme believe that the large number of neurons contain 

15 a distribution among them that allows the various 
dimensions of sense to be recognized so that perception 
can take place. However, it has not been possible to 
determine how these various dimensions are recognized 
using this population coding scheme. It has not been 

20 possible to develop a prosthetic device using this 
population coding principle which can sense an external 
stimulus and transmit information to the brain so that 
meaningful sensory perception can take place. 

SUMMARY OF THE INVENTION 

25 In view of the above, it is an object of the present 

invention, among others, to provide an apparatus and 
method for transmitting prosthetic information to the 
brain. 

It is another object of the present invention to 
30 provide an apparatus and method that will allow 
meaningful encoding of visual, audile, or tactile sensory 
information. 

It is still another object of the present invention 
to provide an apparatus and method to produce simulated 



SUBSTITUTE SHEET 



wo 90/00912 



PCT/LS89/03070 



neural impulses that replicate the neural impulses 
produced naturally by neurons in the body. 

In view of these objects, and others, the present 
invention provides an apparatus and method for 
5 transmitting prosthetic information to the brain in the 
form of simulated neuron impulses that contain a time 
varying component. This time varying component 
corresponds to the time varying component that exists 
during consecutive neuron impulses in an individual 
10 neuron and allows information concerning multiple 
dimensions of a- sense to be contained in the simulated 
neuron impulse train. 

The apparatus and method of the present invention 
sense an external stimulus with an array of sensors. The 
15 output of each of these sensors is used to determine 
simulated neuron impulses associated with each sensor. 
Each sensor acts as a channel that sends its simulated 
neuron impulses to the appropriate sensory location so 
that the sense can be perceived. The array of channels 
20 transmits simulated neuron impulses in parallel to allow 
sensory perception. 

The present invention utilizes a series of 
characteristic sensory functions in combination with 
respective temporal neural filters. Processing of 
25 sensory perceived parameters via these characteristic 
functions and neural filters results in simulated neural 
impulses, or a spike train, containing the properly time 
varying components that allow the brain to sense the 
external stimulus. 
30 The present invention advantageously allows a person 

to perceive environmental parameters, such as light, 
sound, or touch, via simulated neural spike trains that 
emulate naturally occurring spike trains with a temporal 
modulation scheme in such a way that the perception based 
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on the simulated spike train will be functionally related 
to the external stimuli or parameter. 

Other objects and further scope of applicability of 
the present invention will become apparent from the 
5 detailed description to follow, taken in conjunction with 
the accompanying drawings, in which like parts are 
designated by like reference characters. 

BRIEF DESCRIPTION OF THE DRAWING 
FIG. 1 is a partial schematic block diagram of a 
10 system for generating a simulated neural impulse for 
stimulating the- visual cortex of the human brain; 

FIG. 2 is a partial schematic block diagram of the 
system of the present invention; 

FIG. 3 is a legend indicating the manner by which 
15 FIGS. 1 and 2 are to be read; 

FIG. 4 is a graphical illustration of a first neural 
transform in which the abscissa represents time and the 
ordinate represents magnitude; 

FIG. 5 is a table providing quantitative data as to 
20 the neural transform of FIG. 4 in which the left column 
represents time in milliseconds from zero to 315 ms. and 
the right column presents the corresponding magnitude 
value ; 

FIG. 6 is a graphical illustration of a second neural 
25 transform in which the abscissa represents time and the 
ordinate represents magnitude; 

FIG. 7 is a table providing quantitative data as to 
the neural transform of FIG. 6 in which the left column 
represents time in milliseconds from zero to 315 ms. and 
30 the right column presents the corresponding magnitude 
value; 

FIG. 8 is a graphical illustration of a third neural 
transform in which the abscissa represents time and the 
ordinate represents magnitude; 
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FIG. 9 is a table providing quantitative data as to 
the neural transform of FIG. 8 in which the left colusm 
represents time in milliseconds from zero to 315 ms. and 
the right column presents the corresponding magnitude 
5 value ; 

FIG. 10 is a schematic block diagram of a stored- 
program controlled processor for effecting various 
transformations in a digital manner; 

FIG. 11 is a flow diagram illustrating a control 
10 secpience for converting digital values into a temporally 
modulated spike train suitable for stimulating the 
striate cortex to effect, a visual perception in a 
subject; 

FIG. 12 is an illustration of an image pattern, its 
15 corresponding temporal encoding, and its corresponding 
temporally modulated spike train suitable for stimulating 
the visual cortex to provide a perception that is a 
function of the image pattern; and 

FIG. 13 is an illustration of another image pattern, 
20 its corresponding temporal encoding, and its 
corresponding temporally modulated spike train suitable 
for stimulating the visual cortex to provide a perception 
that is a function of the image pattern. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

25 An exemplary system in accordance with the present 

invention for generating simulated neural impulses from 
an environmental parameter and providing a functionally 
related * spike' train for stimulating the brain to effect 
a functionally related perception of the environmental 

30 parameter is shown in FIGS. 1 and 2 and is designated 
generally therein by the reference character 10. The 
system 10 is designed to sense an environmental parameter 
in the form of variations in light that define an image 
and provide a corresponding spike train that is 
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temporally modulated, i.e., pulse position modulated, and 
which, when used to stimulate a portion of the brain of 
a subject, produces a perception that is functionally 
related to the sensed light or image. As shown in 
5 FIGi 1, the exemplary embodiment of the system 10 
includes an array 8 of individual photoreceptors 14. In 
FIG. 1, sixteen photoreceptors 14 are shown in solid-line 
illustration and represent an input sub-array 12 for one 
sub-image information channel for transducing 
10 environmental light into corresponding electrical signals 
presented on respective output lines, indicated generally 
at 18. As can be appreciated, other photoreceptors 14 
are similarly arrayed as the input transducers for other 
sub-image information channels (not specifically shown in 
15 FIGS. 1 and 2) so that a multi-channel system is provided 
to effectively assemble a set. of output spike trains that 
are representative of the image sensed by the array 12. 
The photoreceptors 14 can be individual planar 
photocells, photodiodes, or phototransistors arranged in 
20 an array or can take the form of a subset of the 
photoreceptors on a planar integrated device. If 
desired, an optical system can be provided to image 
subjects of principal and secondary interest within a 
selected field of view onto the array 12. 
25 A scale-factor conditioner 20 is provided to accept 

the outputs of the array 12 and introduce a scaling 
factor that is a function of the light-to-signal 
input/output function of the particular photoreceptors 
14. The scaling factors are determined empirically and 
30 vary, for example, as a function of the sensitivity and 
spectral response of the particular photoreceptors 14. 
in addition, the scale-factor conditioner 20 can 
introduce a logarithmic function (typically log.) to 
compensate for the functional response characteristics of 
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the photoreceptors 14. Where the array 12 is part of a 
scanning camera, for example, the camera typically 
provides the logarithmic function compensation. 

The appropriately scaled output of the scale^factor 
5 conditioner 20 is provided via an appropriate bus 22 
connection to each of three signal-weighting filters 24, 
26, and 28. Each of the signal-weighting filters accepts 
all the scale-factored signal outputs of the scale-factor 
conditioner 20 and multiplies the respective outputs by 

10 a specific weighting value to provide a spatial filter 
effect that is related to respective neural transforms 
described more fully below. For the sixteen signal lines 
of the input sub-array 12, weight values for the signal- 
weighting filters 24, 26, and 28 for the preferred 

15 embodiment are as listed, respectively, in Tables I, II, 
and III below and presented in the four-by-four array 
format corresponding to the input sub-array 12. 

Table I 

-0.423000 -0.004000 -0.118000 -0.099000 
20 -0'.329000 3.366000 0.888000 -0.285000 

0.343000 8.327000 1.527000 -0.297000 

-0.287000 -0.706000 -0.210000 -0.474000 

Table II 

-0. 028300 -0 . 139600 -0 . 029400 -0 . 178400 
25 -0.377900 -0.725700 -0.243300 0.083600 
-0.273900 1.861400 -0.184400 -0.168000 

-0.156200 -0.367900 -0.218200 -0.185200 

Table III 

-0.028640 0.128490 0.155270 -0.033330 

30 0.108840 0.159940 0.183120 0.350690 

-0.081850 0.212080 0.082500 -0.007050 

-0.066830 -0.180620 -0.005260 0.068260 

After the signal weighting is effected by the signal- 
weighting filters 24, 26, and 28, the respective outputs 
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are provided to svunming amplifiers 30, 32, and 34, which 
sum the individual weighted inputs to provide a single 
respective scalar output S that is provided to respective 
cubic function (i.e., = as^+bs^+cS+d) generators 36, 38, 
5 and 40 that scale each signal in a non-linear manner to 
compensate for low input signal values. The summing 
amplifiers can be fabricated from conventional 
operational amplifiers configured in a summing mode, and 
the cubic function generators 36, 38, and 40 can be 

10 fabricated from conventional analog multiplier devices* 
Sample-and-hold circuits 42, 44, and 46 are provided 
respectively at the outputs of the cube function 
generators 36, 38, and 40 and are designed to 
continuously sample the output voltage values. The 

15 system 10 is designed to generate output simulated neural 
impulses on a f ramer by- frame basis, with the simulated 
neural impulses generated subsequent to each frame and 
prior to the stibsecpient frame. A frame is terminated on 
a system-wide basis by a timing signal T=Tf which, when 

20 applied to the sample-and-hold circuits 42, 44, and 46, 
causes the respective circuit to store the signal value 
at its input for subsec[uent processsing as explained 
below. Each sample-and-hold circuit 42, 44, 46 can be 
defined by a series-connected capacitor and switch, such 

25 as a MOSFET. 

Neural transform function units 48, 50, and 52 are 
provided, respectively, at the outputs of the sample-and- 
hold circuits 42, 44, and 46. Each neural transform 
function unit 48, 50, and 52 includes an empirically 

30 determined, time-dependent neural transfer function that 
is multiplied by the value stored in the respective 
sample-and-hold circuits 42, 44, and 46. The transforms 
are determined by measuring biological neuron responses 
to mathematically complete sets of stimuli, e.g., Walsh 
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patterns. » has b.en .mplridly d.t.r»ined «»t the 
transfer functions presented in neural transfer, function 
Tnits 48, 50, ana S. are a statistically ^'^^^J'^^^^. 
actual neural responses. These waveforms, along with the 
corresponding three signal-weighting filters 24, 26, and 
Z th!s allow the syste. 10 to generate siBUlated neural 
i^ulses for use in sti-ulating the visual cort«c of the 

""'^he neural waveform for the neural "•'•'o"""" " 
is reproduced on an enlarged scale in PIC. 4 in which th. 
abscissa represents time, and the ordinate "Present, 
magnitude. The neural waveform of FIG. 4 xs also 
presented in a guantltative manner in FIG. 5 wi«. the 
Lft column representing time In milliseconds between 
zero and 315 ms. and the right column representing the 
corresponding magnitude. In a similar manner, ^" "'^l 
waveform of neural waveform units 50 and 52 are shown in 
enlarged scale in FIGS. 6 and 8, respectively, and their 
Quantitative data in FIGS. 7 and 9. while 64 time and 
Lrespcnding data values are shown in FIGS. 
it is preferred that data values at one millisecond 
intervals be utilized in the signal processing, these 
values provided by interpolation of the data values 
presented in FIGS. 5, 7, and 9. . . „„ 

Further disclosure regarding the neural waveforms and 
the neurophysiology of the present invention is provid^ 
in the following publications, the disclosures of which 
are incorporated herein by refMrence: Richmond, B. J., 
L. M. optican, M. Podell, and H. Splt«r 1987) 
, -Temporal encoding of two-dimensional patterns by single 
units in primate inferior temporal cortex, I. Response 
characteristics." J. Wfl^.TOPtlVSiBl.^ vol. 57:132-146! 
Richmond, B. J. and L. M. Optican (Jan., 1987, "Temporal 
encoding of two-dimensional patterns by single units in 
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primate inferior temporal cortex; II. Quantification of 
response waveform." J. Neurophvsiol. vol. 57:147-161; 
Optican, L. M. and B. J. Richmond (Jan., 1987) "Temporal 
encoding of two-dimensional patterns by single units in 
5 primate inferior temporal cortex. III. Information 
theoretic analysis." J, Neurophvsiol. vol. 57:162-178; 
Richmond, B, J. and L. M. Optican (1986) "Temporal 
encoding of pictures by striate neuronal spike trains. 
I. The multiplex-filter hypothesis." Socle tv for 
10 Neuroscience. Abstract 12:431; Optican, L. M. and B. J. 
Richmond (1986) "Temporal encoding of pictures by striate 
neuronal spike trains. II. Predicting complex cell 
responses." Society for Neuroscience . Abstract 12:431. 
Once the time-dependent neural transfer functions in 
15 the neural waveform function units 48, 50, and 52 are 
multiplied by the values stored in the respective sample- 
and-hold circuits 42, 44, and 46. The products are 
summed in a summing amplifier 54 to provide a time- 
varying output that is related to the image sensed by the 
20 input stib-array 12. The output of the summing amplifier 
54 is provided to an analog-to-digital converter 56 to 
provide a digitized output thereof for . subsequent 
conversion into a spike train as explained more fully 
below. 

25 The neural waveform units 48, 50, and 52 are shown in 

functional block form in FIGS. 1 and 2 and are preferably 
implemented in a digital manner with a stored-program 
controlled processor. For example and as shown in FIG. 
10, the functions performed by the neural waveform units 

30 48, 50, and 52 and the sximming amplifier 54 are performed 
by a microprocessor 100 that is coupled to a read-only- 
memory (ROM) 102 and a random-access-memory (RAM) 104. 
A multiplexer 106 accepts the inputs from the sample-and- 
hold circuits 42, 44, and 46 and, in response to 
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appropriate 'select' signals, provides the selected 
sample-and-hold value to an analog-to-digital (A/D) 
coverter 108 which, in turn provides the corresponding 
digitized sample-and-hold value to the microprocessor 
5 100. A clock 110 provides the necessary timing and 
control signals to the various devices to effect 
synchronized operation. 

The ROM 102 includes the data tables of FIGS. 5, 7, 
and 9. The clock 110 provides the timing and control 

10 signal to the multiplexer 106 to select one of the 
sample-and-hold values which is digitized and stored in 
one of the general purpose registers in the 
microprocessor 100 with the procedure repeated until 
three digital values representative of the three sample— 

15 and-hold values are stored in digital form. The 
microprocessor 100 then reads the first time value (i.e., 
t=0 ms.) from the ROM 102 for a first of the neural 
waveforms and effects a multiplication by the 
corresponding digitized value in sample-and-hold 42 and 

20 stores the respective product temporarily in the RAM 104. 
In a similar manner, the first values for the second and 
the successive neural waveforms are obtained from the ROM 
102, multiplied by their corresponding value from their 
respective sample-and-hold circuits, and the products 

25 stored. The three products are then digitally summed and 
the results again stored. The second time value for the 
various neural waveforms are then obtained, multiplied by 
their respective digital sample-and-hold values, and the 
sum thereof stored in the RAM 104. As can be 

30 appreciated, the process is repeated in a recurring 
manner until a set of output values is obtained that 
corresponds to the output of the analog-to-digital 
converter 56 of the functional block diagram of FIGS. 1 
and 2. Hence, FIG. 10 provides a circuit that implements 
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functional blocks 48^ 50, 52/ 54, and 56 of FIGS. 1 and 
2. • 

Once the neural waveforms are multiplied by their 
respective sample-and-hold values and the products sumiaed 
5 and digitized, as shown in functional form in FIGS. 1 and 
2. or by the processor* implementated form in FIG. 10, the 
digital output is converted to a spike train the spacing 
of which encodes information functionally related to that 
sensed by the input sub-array 12. As shown in FIG. 2, 

10 the digital output of the analog-to-digital converter 56, 
which represents a succession of digital values 
representative of the processed neural waveform values, 
is presented to a stored-program . controlled 
microprocessor 58 that is coupled to read-only-memory 

15 (ROM) 60 that contains a program sequence, as presented 
in FIG. 11, and a random-access-memory (RAM) 62 for 
storing various intermediate and other values. A clock 
64 provides timing pulses while a controller 66 provides 
the necessary select, enable, and control signals to 

20 provide synchronous operation of the various devices. 
The microprocessor 58 is coupled, along with appropriate 
control lines, to a counter 68 that is periodically 
parallel-loaded with a preset threshold value X from a 
register 70. The output of the counter 68 is provided to 

25 a one-shot monostable multi-vibrator 72 that provides, in 
response to an appropriate trigger signal, a pulse or 
spike of selected pulse amplitude and duration. As 
explained below, a succession of such spikes are provided 
with an inter-spike temporal spacing that contains the 

30 encoded information from the image sensed by the input 
sub-array 12. The output of the one-shot 72 is provided 
through a conditioning amplifier 74 which controls the 
voltage output to provide a spike train of appropriate 
voltage level (typically 10 to 500 microvolts with a 50 
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to 100 microsecond spike width duration) for stimulating 
the appropriate spot or area of the visual cortex. 

The microprocessor 58 of FIG, 2 operates in . 
accordance with the control sequence of FIG. 11 to 
5 produce a temporally modulated spike train as a fuinction 
of the digital values presented from the analog-to- 
digital converter 56. As shown in FIG. 11, the system 
initializes, in part, by storing 1 to the data pointer 
variable, N, with N having a maximum value Nmax that 
10 represents the total number of digital data values for 
that frame and which must be converted into a spike train 
representative of that frame. For example, a frame can 
be represented by N^ax ~ digital values determined by 
interpolation of the 64 times values shown in FIGS. 5, 7, 
15 and' 9, each value of which is the sum of the products of 
the corresponding values of the neural waveforms (FIGS., 
4-9) and the sample-and-hold values. 

After initialization, the one-shot 72 is triggered to 
provide an initial spike, have a pulse duration or spike 
20 width of between 50 and 100 microseconds and a selected 
voltage amplitude. A query is presented to determine if 
all the digital data values for a frame have been 
processed, i.e., if N = N„ax# and, if so, the control 
returns to re-initialize for the next frame. If less 
25 than all the digital data values have been processed, the 
N^*^ digital data value is added to the contents of the 
counter 68. A query is presented to determine if the 
accumulated contents of the counter 68 are greater than 
a selected value X (i.e., 1000), and, if not, the data 
30 pointer N is incremented by 1 and the program loops to 
add the next data value to the counter 68, provided that 
N is not equal to N,„ax* counter sum is greater that. 

X, the counter is decremented by X by effecting a 
parallel load from the register 70, and, substantially 
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concurrently, triggering the one- shot 72 to produce the 
next successive spike. Thereafter, the data pointer N 
is incremented by one with the program looping to 
continue to successively add the digital data values to 
5 the counter 68, decrement by X when X is exceeded, and 
issue the next successive spike. As can be appreciated, 
the time duration between each spike is a fxinction of the 
rate at which the digital value accumulates to exceed X 
so that a temporally modulated spike train is generated, 

10 i.e., a spike train in which the inter-spike timing is 
varied as a function of the products of the corresponding 
values of the neural waveforms (FIGS. 4-9) with the 
sample-and-hold values that are a function of the image 
sensed by the input sub*array 12. 

15 The spike train output of the one*shot 72 is provided 

to an amplifier 74 which provides a buffered output, 
typically no more than 10 - 100 microvolts, that is 
provided to a neural probe that is placed in a selected 
location on an appropriately mapped visual cortex of the 

20 brain of a sxibject to stimulate a neuron (s) to cause 
perception of an image or light pattern that is 
functionally related to that sensed by the input sub- 
array 12. The other adjacent channels (riot specifically 
shown in FIGS. 1 and 2) similarly drive other neural 

25 electrodes place in an electrode grid pattern 
corresponding to the visual maps of the entire input 
array 8. 

Exemplary temporally modulated spike trains, their 
corresponding waveforms, and the corresponding Walsh 
30 patterns are shown in FIGS., 12 and 13, and, as shown, 
the inter-spike spacing and the rate of changes thereof 
varies as a function of the waveform to produce a 
temporally modulated spike train. In practice, it has 
been found that inter-spike spacing varies between one 
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and 300 milliseconds. 

The present invention has been disclosed as useful 
in providing a simulated spike train responsive to a 
sensed optical image; as can be appreciated, a different 
5 input sensor, such as an array of sound-responsive 
devices or pressure responsive devices can likewise be 
utilized to provide a simulated spike train responsive to 
a sensed sound or pressure. The present invention 
advantageously allows a person to perceive environmental 

10 parameters via temporally modulated simulated neural 
spike trains that emulate naturally occurring spike 
trains in such a way that a perception based on the 
simulated spike train will be functionally related to the 
external stimuli or parameter. 

15 AS will be apparent to those skilled in the art, 

various changes and modifications may be made to the 
illustrated apparatus and method for transmitting 
prosthetic information to the brain of the present- 
invention without departing from the spirit and scope of 

20 the invention as determined in the appended claims and 
their legal equivalent. 
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Vlhat, is claimed is: 

1. An apparatus for transmititing prosthetic 
information about an external stimulus to the brain . 
comprising: 

5 means for sensing said external stimulus; 

means for converting said sensed external stimulus 
into simulated neuron impulses that can contain 
information about multiple dimensions of said sensed 
esctemal stimulus due to the presence of a time varying 
10 component in said simulated neuron signals; and 

means for transmitting said simulated neuron impulses 
to the brain. 

2. A method for transmitting prosthetic information 
about an external stimulus to the brain comprising the 

15 steps of: 

sensing said external stimulus; 

converting said sensed external stimulus into 
simulated neuron impulses that can contain infozmation 
about multiple dimensions of said sensed external 
20 stimulus due to the presence of a time varying component 
in said simulated neuron signals; and 

transmitting said simulated neuron impulses to the 
brain. 
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